Annotating the Legacy Web with Lixto

Introduction The Semantic Web is still a vision. The unstructured Web of today contains millions of documents which cannot be queried and where layout and structure are heavily mixed. Moreover, they are not annotated at all. There is a huge gap between Web information and the qualified, structured data as required in corporate information systems. According to the vision of the Semantic Web, all information available on the Web will be suitably structured, annotated, and qualified in the future. However, until this goal is reached, and also, towards a faster achievement of this goal, relevant data can be (semi-)automatically extracted from HTML documents and automatically translated into a structured format, e.g., XML. A program that automatically extracts data and transforms it into another format (markups the content with semantic information) is called a wrapper. Intelligent content extraction provides the foundation for automatic generation of semantic markup. Various approaches to automatic content extraction have been proposed, ranging from machine learning techniques to pattern recognition techniques. However, these approaches in general fail to produce useful results due to the complexity of Web pages. Other approaches suggest the manual editing of script files that wrap the relevant data from Web pages into more structured formats. Such processes are time-consuming, hard to understand for non-technical wrapper designers, and script files are not easy to maintain. We propose another approach a supervised and visual definition of content extraction. Based on interactively identifying and extracting relevant parts of HTML documents and translating content to XML format, we designed and implemented the efficient wrapper generation tool Lixto Visual Wrapper [2] which is well-suited for building HTML to XML wrappers. Such a wrapper can be applied to continually extract relevant information from this class of Web pages. The process of wrapping consists in two steps: First, the identification phase, where relevant fragments of Web pages are extracted. Such extraction rules are semi-automatically specified by a wrapper designer (e.g., in a purely visual way in Lixto Visual Wrapper). This step is followed by the structuring phase where the extracted data is mapped to some destination format, e.g., enriching it with XML tags. With respect to the Semantic Web, a third phase is required: Each information unit needs to be put into relation with other pieces of information. Additionally, methods for automatically relating extracted information to existing domain ontologies are required. The aspect of integration of extracted data plays an important role in querying the Web; data schemes, metadata, and background knowledge such as ontologies must be integrated to reach a uniform semantic interpretation of information. Lixto Transformation Server [6] covers the integration, annotation and delivery process as part of the Lixto set of tools. As long as the Web is not a source of machine-readable data but still merely accessible for human beings, wrapper technology provides the possibility for computers to query and interact with it [3].